Dynamic Mental Models in Weather Forecasting

There are many definitions, descriptions, and usages of the term “mental model.” Frequently, the definition of mental model is not described, leaving what the author means as an exercise for the reader. I propose a very explicit definition for a dynamic mental model and then show how that definition can be applied in the domain of meteorological forecasting. Specifically, I suggest that a dynamic mental model is a mix of images and propositions, relies on qualitative and spatial relationships, allows dynamic, runnable results to be inspected, and results in an inference. Finally, I offer suggestions on how to improve the usefulness of the term mental model. Many meteorologists and cognitive scientists have suggested that meteorologists use a “mental model” to predict the weather (Hoffman, Coffey, & Ford, 2000; Lowe, 1994; Perby, 1989; Trafton et al., 2000). However, the term “mental model” has been used by many different researchers to mean many different things. For example, mental models can be based on visual images or abstract situations representing a single possibility (Byrne, 2004). These mental models have been used to study reasoning and deduction (Johnson-Laird & Byrne, 1991). Norman uses the term mental model to explore how devices like computers, ATMs, etc. work (Norman, 1983, 1988). Gentner describes a mental model as ``a representation of some domain or situation that supports understanding, reasoning, and prediction'' (Gentner, 2001) and has used mental models to explore how dynamic systems unfold (among other things). In general, the scientific community uses the term “mental model” to mean one of several mental representations; additionally, it may have one of several different properties. If meteorologists use mental models to help them predict the weather, what exactly does that mean, and what is the evidence for their using mental models? Instead of performing an exhaustive review of how different researchers have used the term mental model, I will instead describe the definition that our group has been using the last few years and provide supporting evidence for each aspect of that definition. In general, this paper’s description of mental model will come closest to that used by Gentner and Stevens (1983). Not coincidentally, many weather forecasters use a very similar (though equally fuzzy) definition (e.g., Perby, 1989). Our definition of a mental model will be: • a mix of images and propositions • relying on qualitative and spatial relationships • allowing dynamic, runnable results to be mentally inspected • resulting in an inference • almost always requiring a great deal of domain knowledge Evidence for each of these points will be discussed in the sections below. A mental model is primarily a spatial representation, and this spatial information is represented qualitatively not metrically or quantitatively. It is a mix of images and propositions because, while most researchers see the importance of the imagistic component, it must be able to connect to our propositional reasoning mechanisms (Altmann & Trafton, 2002; Anderson, Conrad, & Corbett, 1989). It also must be “runnable” (dynamic) – the process that makes a mental model unique from many other representations. Finally, a mental model frequently requires a great deal of thought and energy to create compared to other mental operations; the output of a completed mental model run is an inference. How well does this definition match to empirical work in the meteorological forecasting domain? Recent work suggests that meteorologists do, indeed, form a mental model while making a forecast and use that mental model to predict what the weather will be. A mix of images and propositions Recent empirical work suggests that weather forecasters do, indeed use a mix of images and propositions to make a forecast. Many cognitive psychologists believe there is a strong propositional component to the way that we think: there are multiple computational cognitive models that rely exclusively on propositional accounts (Anderson & Lebiere, 1998; Kieras & Meyer, 1997; Laird, Newell, & Rosenbloom, 1987; Newell, 1990). However, many cognitive psychologists equally believe that much of cognition has a strong pictorial or imagistic component (Kosslyn, 1980, 1990; Laeng & Teodorescu, 2002; Thomas, 1999). In the forecasting domain, there seems to be a strong belief that weather forecasters create a mental image in their head of what the weather is and what it will be. It is not clear, of course, whether forecasters’ mental image is simply epiphenomenal to the actual processes or whether it is the imagistic representation that is doing the “work” (e.g., Pylyshyn, 1973). The evidence from forecasters that suggests that working meteorologists have a mental model is typically based on idiosyncratic introspection and/or leading questions. The work is thus not methodologically strong from a human-factors point of view, but it should be noted that most forecasters truly believe they keep a picture in their head about the weather. Relying on qualitative and spatial relationships Evidence for the qualitative nature of mental models comes from both computational models and systems and empirical evidence of forecasters. Forbus has virtually created an entire sub-field of AI on qualitative physics (Forbus, 199; Forbus & Gentner, 1997) and qualitative reasoning (see, for example, the Winter 2004 issue of AI Magazine, which devoted eight articles to qualitative reasoning). However, what is the evidence that forecasters use a qualitative representation to help them forecast the weather? This issue is especially relevant because the visualizations that a forecaster examines shows quantitative information to be extracted, a forecaster frequently makes a quantitative prediction about what the weather will be (e.g., the temperature will be 72°), and, meteorology is considered a “hard” science with a great deal of physics and mathematical training traditionally needed. The above figure is a traditional meteorological forecasting visualization – it shows temperature (color coded), sea-level pressure (iso-lines with numeric labels), wind-speed and wind-direction (wind-barbs) overlayed over most of the United States. Even though this visualization displays quantitative data, several studies of expert forecasters have shown that forecasters actually extract primarily qualitative information! For example, a forecaster looking at a visualization may say something like “the temperature in Canada is particularly cool” rather than saying “the temperature is 52°.” What is particularly interesting about this example is that the forecaster must then take their qualitative representation (or mental model) and create a numeric forecast from it. Again, forecasters are able to perform this task quite accurately. The evidence for this “turning pictures into numbers” process occurs, we argue, because forecasters create a mental model that connects the visualization, their expertise and knowledge of various weather systems, and the reasoning processes (especially the dynamics, discussed in the next section). The basic effect of extracting qualitative information and generating quantitative information has been shown with Navy forecasts (Trafton et al., 2000) and replicated with Australian meteorologists (Kirschenbaum, Trafton, & Kramer, in press). Allowing dynamic results to be mentally inspected One of the least controversial components of a mental model is that it is a dynamic representation that allows the “playing out” of different hypothetical situations. For example, if a forecaster is unsure exactly where a front may be, she may mentally play out the previous location of the front (e.g., mentally animate the progression and movement based on its history) and decide based on her own understanding of the current weather situation where it should be. After she has determined the location of the front, she must also determine how other variables will be affected by the front – temperature, wind speed and direction, precipitation, etc. This variable propogation seems to happen dynamically. Additionally, the mental dynamics of weather forecasters as they predict the weather has been shown in both laboratory studies (Lowe, 1994, 1999) and more naturalistic studies (Bogacz & Trafton, 2002). Additionally, other researchers have shown the prevalence of dynamic reasoning in other domains including mechanical diagram understanding (Hegarty, 1992; Hegarty & Sims, 1994) and scientific reasoning (Trickett & Trafton, under review). Bogacz and Trafton (2002) showed that experienced Naval forecasters mentally animated static meteorological visualizations. Interestingly, the forecasters examined very few dynamic visualizations (e.g., satellite loops) even though they had access to animations, suggesting that forecasters would rather mentally animate their own weather models than see an explicit animation. Resulting in an inference A forecaster’s weather prediction is, by definition, an inference: no one knows with complete certainty what the weather will be in the future. Computational weather models make extremely precise quantitative predictions, but, as shown above, forecasters do not rely solely on the numeric results of individual forecasts, using instead the qualitative values and relationships between different variables. Trafton and his colleagues have shown that forecasters combine the results of multiple sources of information to make each quantitative prediction rather than simply reading off a single numeric answer. For example, a forecaster may examine several different information sources and extract the specific temperature over a location: 78°, 74°, and 82°. The forecaster may make a prediction that the temperature will be 76°. Note that 76° is not a numerical average, though it is within the range of values. This number is an inference about what the weather will be. Our claim is that the forecaster has built a mental model, run it through several possibilities, and created the most likely scenario based on his experience to create a prediction for what the temperature will be. A reliance on domain knowledge In most cases, a mental model is created with a great deal of domain knowledge: most mental models are created in domains where a lot of knowledge is critical to solve problems. In meteorology this is certainly the case: meteorologists need a great deal of information to create accurate forecasts. One of the open questions about expertise and meteorology is how much of a mental model is created and used by novice or journeyman forecasters. Summary and Discussion This paper has suggested that a mental model is a mix of images and propositions that consists of both qualitative and spatial relationships. A mental model allows dynamic, runnable results to be mentally inspected and results in an inference. Finally, a mental model usually comes from strong domain knowledge. This definition has been explored and supported in the domain of meteorology. There are several reasons why the definition of mental models is important in the field of human factors. First, imprecise definitions hurt the science of the field – if definitions are not explained, the meaning of the term becomes diluted (as could be argued is happening now with the term mental models). By sticking with a precise definition of the term mental model, researchers can tightly constrain the representational and process assumptions. Additionally, if a particular study does not fit the definition of a mental model, different representational assumptions may be needed, and different processes may need to be assumed in order to fully account for the study. Of primary importance is the point that the term mental model should not be used to describe an ambiguous or unknown representation of a complex task, or that generally means “how something works.” How does a good description of mental model enlighten us about the cognitive processes in meteorology? Another way of asking that question is “Why is knowing that meteorologists use mental models important?” One possible answer is that if there is a mismatch between external information and the user’s mental model-building capabilities, it is going to be much more difficult to build a coherent and robust mental model. For example, by our definition, a mental model is a mix of images and propositions. Currently, meteorologists are able to get both imagistic and textual (propositional) data on the weather. Second, a mental model relies on qualitative and spatial relationships. This part of the definition strongly suggests that visualizations should emphasize the qualitative aspect of data display rather than the purely numeric. The wind-barb is an excellent example of a good qualitative glyph: it shows at a glance the direction and speed of wind at that location. Interestingly, the quantitative information can also be extracted from a wind-barb. Since a mental model is dynamic and runnable, information should be presented to facilitate that dynamic quality. Simply providing animations is probably not the best way to support a mental model because a mental model must be run inside the person’s head rather than on-screen. Thus, visualizations should be presented in time-order so that the forecaster can mentally animate it. A mental model also frequently results in an inference. This aspect of a mental model is probably the least understood; there are many ways to facilitate inferences. One possible way to facilitate inferencing within meteorology is to facilitate the comparison process: knowing how different weather models differ from or are similar to each other (and how different they are from recent satellite pictures) may facilitate the overall inferencing process. In summary, researchers should precisely define what they mean by “mental model.” This paper has suggested one possible definition and shown how it maps to meteorology. This paper has also described the advantages of using a mental model approach to mis-matches between external and internal representations. Importantly, any difficulty a person has with a task does not automatically mean there is a mental model mismatch. These mis-matches (based on the definition) can be used to guide not only future research but product development as well. Acknowledgements: Work on this project was supported by ONR grants M12439 and N0001403WX30001 to Greg Trafton. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Navy.